{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 9: Hyper-Parameter Tuning with Cross-Validation\n",
    "___\n",
    "\n",
    "## Exercises\n",
    "\n",
    "**9.1** Using the function getTestData from Chapter 8, form a synthetic dataset of 10,000 observations with 10 features, where 5 are informative and 5 are noise.\n",
    "**(a)** Use GridSearchCV on 10-fold CV to find the C, gamma optimal hyperparameters on a SVC with RBF kernel, where `param_grid={'C':[1E2,1E-1,1,10,100],'gamma':[1E-2,1E-1,1,10,100]}` and the scoring function is neg_log_loss.\n",
    "- **(b)** How many nodes are there in the grid?\n",
    "- **(c)** How many fits did it take to find the optimal solution?\n",
    "- **(d)** How long did it take to find this solution?\n",
    "- **(e)** How can you access the optimal result?\n",
    "- **(f)** What is the CV score of the optimal parameter combination?\n",
    "- **(g)** How can you pass sample weights to the SVC?\n",
    "\n",
    "**9.2** Using the same dataset from exercise 1,\n",
    "- **(a)** Use RandomizedSearchCV on 10-fold CV to find the C, gamma optimal hyper-parameters on an SVC with RBF kernel, where `param_distributions={'C':logUniform(a=1E-2,b=1E2),'gamma':logUniform(a=1E-2,b=1E2)},n_iter=25` and `neg_log_loss` is the scoring function.\n",
    "- **(b)** How long did it take to find this solution?\n",
    "- **(c)** Is the optimal parameter combination similar to the one found in exercise 1?\n",
    "- **(d)** What is the CV score of the optimal parameter combination? How does it compare to the CV score from exercise 1?\n",
    "\n",
    "**9.3** From exercise 1,\n",
    "- **(a)** Compute the Sharpe ratio of the resulting in-sample forecasts, from point 1.a (see Chapter 14 for a definition of Sharpe ratio).\n",
    "- **(b)** Repeat point 1.a, this time with `accuracy` as the scoring function. Compute the in-sample forecasts derived from the hyper-tuned parameters.\n",
    "- **(c)** What scoring method leads to higher (in-sample) Sharpe ratio?\n",
    "\n",
    "9.4 From exercise 2,\n",
    "- **(a)** Compute the Sharpe ratio of the resulting in-sample forecasts, from point 2.a.\n",
    "- **(b)** Repeat point 2.a, this time with `accuracy` as the scoring function. Compute the in-sample forecasts derived from the hyper-tuned parameters.\n",
    "- **(c)** What scoring method leads to higher (in-sample) Sharpe ratio?\n",
    "\n",
    "**9.5** Read the definition of log loss, $L[Y, P]$.\n",
    "- **(a)** Why is the scoring function `neg_log_loss` defined as the negative log loss, $−L[Y, P]$?\n",
    "- **(b)** What would be the outcome of maximizing the log loss, rather than the negative log loss?\n",
    "\n",
    "**9.6** Consider an investment strategy that sizes its bets equally, regardless of the forecast’s confidence. In this case, what is a more appropriate scoring function for hyper-parameter tuning, accuracy or cross-entropy loss?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
